Influence of a Large Language Model on Diagnostic Reasoning: A Randomized Clinical Vignette Study.

Importance: Diagnostic errors are common and cause significant morbidity. Large language models (LLMs) have shown promise in their performance on both multiple-choice and open-ended medical reasoning examinations, but it remains unknown whether the use of such tools improves diagnostic reasoning. Objective: To assess the impact of the GPT-4 LLM on physicians' diagnostic reasoning compared to conventional resources. Design: Multi-center, randomized clinical vignette study. Setting: The study was conducted using remote video conferencing with physicians across the country and in-person participation across multiple academic medical institutions. Participants: Resident and attending physicians with training in family medicine, internal medicine, or emergency medicine. Interventions: Participants were randomized to access GPT-4 in addition to conventional diagnostic resources or to just conventional resources. They were allocated 60 minutes to review up to six clinical vignettes adapted from established diagnostic reasoning exams. Main Outcomes and Measures: The primary outcome was diagnostic performance based on differential diagnosis accuracy, appropriateness of supporting and opposing factors, and next diagnostic evaluation steps. Secondary outcomes included time spent per case and final diagnosis. Results: 50 physicians (26 attendings, 24 residents) participated, with an average of 5.2 cases completed per participant. The median diagnostic reasoning score per case was 76.3 percent (IQR 65.8 to 86.8) for the GPT-4 group and 73.7 percent (IQR 63.2 to 84.2) for the conventional resources group, with an adjusted difference of 1.6 percentage points (95% CI -4.4 to 7.6; p=0.60). The median time spent on cases for the GPT-4 group was 519 seconds (IQR 371 to 668 seconds), compared to 565 seconds (IQR 456 to 788 seconds) for the conventional resources group, with a time difference of -82 seconds (95% CI -195 to 31; p=0.20). GPT-4 alone scored 15.5 percentage points (95% CI 1.5 to 29, p=0.03) higher than the conventional resources group. Conclusions and Relevance: In a clinical vignette-based study, the availability of GPT-4 to physicians as a diagnostic aid did not significantly improve clinical reasoning compared to conventional resources, although it may improve components of clinical reasoning such as efficiency. GPT-4 alone demonstrated higher performance than both physician groups, suggesting opportunities for further improvement in physician-AI collaboration in clinical practice.

'We're like Spider-Man; with great power comes great responsibility': Coaches' experiences supporting struggling medical students.

BACKGROUND: Medical students can experience a range of academic and non-academic struggles. Coaching is a valuable strategy to support learners, but coaches describe working with struggling learners as taxing. Transformative learning theory (TLT) provides insights into how educators grow from challenging experiences to build resilience. This study explores how coaches evolve as educators through supporting struggling students. METHODS: This qualitative study grounded in an interpretivist paradigm used interviews of longitudinal medical student coaches at two academic institutions. Interviews, using TLT as a sensitizing concept, explored coaches' experience coaching struggling learners. We performed thematic analysis. RESULTS: We interviewed 15 coaches. Coaches described supporting students through multi-faceted struggles which often surprised the coach. Three themes characterized coaches' experiences: personal responsibility, emotional response, and personal learning. Coaches shouldered high personal responsibility for learners' success. For some, this burden felt emotional, raised parental instincts and questions about maintaining boundaries with learners. Coaches evolved their coaching approach, challenged biases, and built skills. Coaches learned to better appreciate the learner point of view and employ resources to support students. DISCUSSION: Through navigating learner struggles, educators can gain self-efficacy, learn to understand learners' perspectives, and evolve their coaching approach to lessen their personal emotional burden through time.

Lessons in clinical reasoning - pitfalls, myths, and pearls: a case of tarsal tunnel syndrome caused by an intraneural ganglion cyst.

OBJECTIVES: Intraneural ganglionic cysts are non-neoplastic cysts that can cause signs and symptoms of peripheral neuropathy. However, the scarcity of such cases can lead to cognitive biases. Early surgical exploration of space occupying lesions plays an important role in identification and improving the outcomes for intraneural ganglionic cysts. CASE PRESENTATION: This patient presented with loss of sensation on the right sole with tingling numbness for six months. A diagnosis of tarsal tunnel syndrome was made. Nerve conduction study revealed that the mixed nerve action potential (NAP) was absent in the right medial and lateral plantar nerves. The magnetic resonance imaging (MRI) found a cystic lesion measuring 1.4×1.8×3.8 cm as the presumed cause of the neuropathy. Surgical exploration revealed a ganglionic cyst traversing towards the flexor retinaculum with baby cysts. The latter finding came as a surprise to the treating surgeon and was confirmed to be an intraneural ganglionic cyst based on the histopathology report. CONCLUSIONS: Through integrated commentary by a case discussant and reflection by an orthopedician, this case highlights the significance of the availability heuristic, confirmation bias, and anchoring bias in a case of rare disease. Despite diagnostic delays, a medically knowledgeable patient's involvement in their own care lead to a more positive outcome. A fish-bone diagram is provided to visually demonstrate the major factors that contributed to the diagnostic delay. Finally, this case provides clinical teaching points in addition to a pitfall, myth, and pearl related to availability heuristic and the sunk cost fallacy.

Development and Validation of a Formative Assessment Tool for Nephrology Fellows' Clinical Reasoning.

BACKGROUND: Diagnostic errors are commonly driven by failures in clinical reasoning. Deficits in clinical reasoning are common among graduate medical learners, including nephrology fellows. We created and validated an instrument to assess clinical reasoning in a national cohort of nephrology fellows and established performance thresholds for remedial coaching. METHODS: Experts in nephrology education and clinical reasoning remediation designed an instrument to measure clinical reasoning through a written patient encounter note from a web-based, simulated AKI consult. The instrument measured clinical reasoning in three domains: problem representation, differential diagnosis with justification, and diagnostic plan with justification. Inter-rater reliability was established in a pilot cohort ( n =7 raters) of first-year nephrology fellows using a two-way random effects agreement intraclass correlation coefficient model. The instrument was then administered to a larger cohort of first-year fellows to establish performance standards for coaching using the Hofstee method ( n =6 raters). RESULTS: In the pilot cohort, there were 15 fellows from four training program, and in the study cohort, there were 61 fellows from 20 training programs. The intraclass correlation coefficients for problem representation, differential diagnosis, and diagnostic plan were 0.90, 0.70, and 0.50, respectively. Passing thresholds (% total points) in problem representation, differential diagnosis, and diagnostic plan were 59%, 57%, and 62%, respectively. Fifty-nine percent ( n =36) met the threshold for remedial coaching in at least one domain. CONCLUSIONS: We provide validity evidence for a simulated AKI consult for formative assessment of clinical reasoning in nephrology fellows. Most fellows met criteria for coaching in at least one of three reasoning domains, demonstrating a need for learner assessment and instruction in clinical reasoning.

Framework and Schema are False Synonyms: Defining Terms to Improve Learning.

Clinical reasoning is an essential expertise of health care professionals that includes the complex cognitive processes that lead to diagnosis and management decisions. In order to optimally teach, learn, and assess clinical reasoning, it is imperative for teachers and learners to have a shared understanding of the language. Currently, educators use the terms schema and framework interchangeably but they are distinct concepts. In this paper, we offer definitions for schema and framework and use the high-stakes field of aviation to demonstrate the interplay of these concepts. We offer examples of framework and schema in the medical education field and discuss how a clear understanding of these concepts allows for greater intentionality when teaching and assessing clinical reasoning.

Lessons in clinical reasoning - pitfalls, myths, and pearls: a case of crushing, substernal chest pain.

OBJECTIVES: Diagnostic error is not uncommon and diagnostic accuracy can be improved with the use of problem representation, pre-test probability, and Bayesian analysis for improved clinical reasoning. CASE PRESENTATION: A 48-year-old female presented as a transfer from another Emergency Department (ED) to our ED with crushing, substernal pain associated with dyspnea, diaphoresis, nausea, and a tingling sensation down both arms with radiation to the back and neck. Troponins were elevated along with an abnormal electrocardiogram. A negative myocardial perfusion scan led to the patient's discharge. The patient presented to the ED 10 days later with an anterior ST-elevation myocardial infarction. CONCLUSIONS: An overemphasis on a single testing modality led to diagnostic error and a severe event. The use of pre-test probabilities guided by history-taking can lead to improved interpretation of test results, ultimately improving diagnostic accuracy and preventing serious medical errors.

A novel virtual course to teach medical students high-value decision-making.

BACKGROUND: Although a clinician's ability to employ high-value decision-making is influenced by training, many undergraduate medical education programmes lack a formal curriculum in high-value, cost-conscious care. We present a curriculum developed through a cross-institutional collaboration that was used to teach students at two institutions about this topic and can serve as a framework for other institutions to develop similar curricula. APPROACH: The faculty from the University of Virginia and the Johns Hopkins University School of Medicine created a 2-week-long online course to teach medical students the fundamentals of high-value care. The course consisted of learning modules, clinical cases, textbook studies, journal clubs and a competitive 'Shark Tank' final project where students proposed a realistic intervention to promote high-value clinical care. EVALUATION: Over two-thirds of students rated the course's quality as excellent or very good. Most found the online modules (92%), assigned textbook readings (89%) and 'Shark Tank' competition (83%) useful. To evaluate the student's ability to apply the concepts learned during the course in clinical contexts, we developed a scoring rubric based on the New World Kirkpatrick Model to evaluate students' proposals. Groups chosen as finalists (as determined by faculty judges) were more likely to be fourth-year students (56%), achieved higher overall scores (p = 0.03), better incorporated cost impact at several levels (patient, hospital and national) (p = 0.001) and discussed both positive and negative impacts on patient safety (p = 0.04). IMPLICATIONS: This course provides a framework for medical schools to use in their teaching of high-value care. Cross-institutional collaboration and online content overcame local barriers such as contextual factors and lack of faculty expertise, allowed for greater flexibility, and enabled focused curricular time to be spent on a capstone project competition. Prior clinical experience amongst medical students may be an enabling factor in promoting application of learning related to high-value care.

The future of teaching management reasoning: important questions and potential solutions.

Management reasoning is distinct from but inextricably linked to diagnostic reasoning in the iterative process that is clinical reasoning. Complex and situated, management reasoning skills are distinct from diagnostic reasoning skills and must be developed in order to promote cogent clinical decisions. While there is growing interest in teaching management reasoning, key educational questions remain regarding when it should be taught, how it can best be taught in the clinical setting, and how it can be taught in a way that helps mitigate implicit bias. Here, we describe several useful tools to structure teaching of management reasoning across learner levels and educational settings. The management script provides a scaffold for organizing knowledge around management and can serve as a springboard for discussion of uncertainty, thresholds, high-value care, and shared decision-making. The management pause reserves space for management discussions and exploration of a learner's reasoning. Finally, the equity reflection invites learners to examine management decisions from a health equity perspective, promoting the practice of metacognition around implicit bias. These tools are easily deployable, and - when used regularly - foster a learning environment primed for the successful teaching of management reasoning.

The Diagnostic Medical Interview.

The diagnostic medical interview spans from the chief concern to the formation of a differential diagnosis. The patient's unique expression of their symptoms is the central component of this conversation. The interview should begin by eliciting the patient's chief concern with an open-ended question and then move through 3 nonlinear phases: open-ended elicitation, guided elicitation, and hypothesis-driven elicitation. Performing a comprehensive medical interview by obtaining background health information and the review of systems can help to expand or shrink the differential diagnosis. Clinicians should obtain information about specific symptoms and background information with a significant likelihood to narrow the differential diagnosis.

REACT: Rapid Evaluation Assessment of Clinical Reasoning Tool.

INTRODUCTION: Clinical reasoning encompasses the process of data collection, synthesis, and interpretation to generate a working diagnosis and make management decisions. Situated cognition theory suggests that knowledge is relative to contextual factors, and clinical reasoning in urgent situations is framed by pressure of consequential, time-sensitive decision-making for diagnosis and management. These unique aspects of urgent clinical care may limit the effectiveness of traditional tools to assess, teach, and remediate clinical reasoning. METHODS: Using two validated frameworks, a multidisciplinary group of clinicians trained to remediate clinical reasoning and with experience in urgent clinical care encounters designed the novel Rapid Evaluation Assessment of Clinical Reasoning Tool (REACT). REACT is a behaviorally anchored assessment tool scoring five domains used to provide formative feedback to learners evaluating patients during urgent clinical situations. A pilot study was performed to assess fourth-year medical students during simulated urgent clinical scenarios. Learners were scored using REACT by a separate, multidisciplinary group of clinician educators with no additional training in the clinical reasoning process. REACT scores were analyzed for internal consistency across raters and observations. RESULTS: Overall internal consistency for the 41 patient simulations as measured by Cronbach's alpha was 0.86. A weighted kappa statistic was used to assess the overall score inter-rater reliability. Moderate reliability was observed at 0.56. DISCUSSION: To our knowledge, REACT is the first tool designed specifically for formative assessment of a learner's clinical reasoning performance during simulated urgent clinical situations. With evidence of reliability and content validity, this tool guides feedback to learners during high-risk urgent clinical scenarios, with the goal of reducing diagnostic and management errors to limit patient harm.

Concordance of Narrative Comments with Supervision Ratings Provided During Entrustable Professional Activity Assessments.

BACKGROUND: Use of EPA-based entrustment-supervision ratings to determine a learner's readiness to assume patient care responsibilities is expanding. OBJECTIVE: In this study, we investigate the correlation between narrative comments and supervision ratings assigned during ad hoc assessments of medical students' performance of EPA tasks. DESIGN: Data from assessments completed for students enrolled in the clerkship phase over 2 academic years were used to extract a stratified random sample of 100 narrative comments for review by an expert panel. PARTICIPANTS: A review panel, comprised of faculty with specific expertise related to their roles within the EPA program, provided a "gold standard" supervision rating using the comments provided by the original assessor. MAIN MEASURES: Interrater reliability (IRR) between members of review panel and correlation coefficients (CC) between expert ratings and supervision ratings from original assessors. KEY RESULTS: IRR among members of the expert panel ranged from .536 for comments associated with focused history taking to .833 for complete physical exam. CC (Kendall's correlation coefficient W) between panel members' assignment of supervision ratings and the ratings provided by the original assessors for history taking, physical examination, and oral presentation comments were .668, .697, and .735 respectively. The supervision ratings of the expert panel had the highest degree of correlation with ratings provided during assessments done by master assessors, faculty trained to assess students across clinical contexts. Correlation between supervision ratings provided with the narrative comments at the time of observation and supervision ratings assigned by the expert panel differed by clinical discipline, perhaps reflecting the value placed on, and perhaps the comfort level with, assessment of the task in a given specialty. CONCLUSIONS: To realize the full educational and catalytic effect of EPA assessments, assessors must apply established performance expectations and provide high-quality narrative comments aligned with the criteria.

Characterization of Clinical Skills Remediation: A National Survey of Medical Schools.

OBJECTIVES: Clinical skills instruction is a standard part of medical school curricula, but how institutions address learners who struggle in this area is less clear. Although recommendations for the remediation of clinical skills at an institutional level have been published, how these recommendations are being implemented on a national scale is unknown. In this descriptive study, we characterize current clinical skills remediation practices at US medical schools and US-accredited Caribbean medical schools. METHODS: We conducted a cross-sectional survey of medical educators who work with struggling students. From March 24, 2020 to April 9, 2020, the Directors of Clinical Skills Remediation Working Group conducted an e-mail survey incorporating four aspects of remediation program design and function: identification, assessment, active remediation, and ongoing evaluation. RESULTS: In total, 92 individuals representing 45 institutions provided descriptive information about their respective remediation programs. The majority of respondents have a formal process of identifying (75%) and assessing (86%) students who are identified as struggling with clinical skills, but lack a standardized method of categorizing deficits. Fewer institutions have a standardized approach to active remediation and ongoing evaluation of struggling learners. Fifty-two percent of institutions provide training to faculty involved in the remediation process. CONCLUSIONS: Although most institutions are able to identify struggling students, they lack a standardized approach to intervene. Remediation effectiveness is limited by a lack of student buy-in and institutional time, expertise, and resources. These findings highlight the need for more formalized structure and standardization in remediation program design and implementation.

Lessons in clinical reasoning ‒ pitfalls, myths and pearls: a case of recurrent pancreatitis.

OBJECTIVES: Cognitive biases can result in clinical reasoning failures that can lead to diagnostic errors. Autobrewery syndrome is a rare, but likely underdiagnosed, condition in which gut flora ferment glucose, producing ethanol. It most frequently presents with unexplained episodes of inebriation, though more case studies are necessary to better characterize the syndrome. CASE PRESENTATION: This is a case of a 41-year old male with a past medical history notable only for frequent sinus infections, who presented with recurrent episodes of acute pancreatitis. In the week prior to his first episode of pancreatitis, he consumed four beers, an increase from his baseline of 1-2 drinks per month. At home, he had several episodes of confusion, which he attributed to fatigue. He underwent laparoscopic cholecystectomy and testing for genetic and autoimmune causes of pancreatitis, which were non-revealing. He was hospitalized 10 more times during that 9-month period for acute pancreatitis with elevated transaminases. During these admissions, he had elevated triglycerides requiring an insulin drip and elevated alcohol level despite abstaining from alcohol for the prior eight months. His alcohol level increased after consumption of complex carbohydrates, confirming the diagnosis of autobrewery syndrome. CONCLUSIONS: Through integrated commentary on the diagnostic reasoning process, this case underscores how overconfidence can lead to premature closure and anchoring resulting in diagnostic error. Using a metacognitive overview, case discussants describe the importance of structured reflection and a standardized approach to early hypothesis generation to navigate these cognitive biases.

A High-Value Care Curriculum Using Individual and Group Structured Reflection.

OBJECTIVE: One-third of all healthcare dollars are wasted, primarily in the form of clinician-ordered unnecessary diagnostic tests and treatments. Medical education has likely played a central role in the creation and perpetuation of this problem. We aimed to create a curriculum for medical students to promote their contribution to high-value care conversations in the clinical environment. METHODS: At a large university medical center between March 2017 and February 2018, we implemented a 3-phase curriculum combining multimodal educational initiatives with individual and group reflection for third-year medical students during their 12-week long Internal Medicine clerkship rotation. Students were asked to identify examples of clinical decision making that lacked attention to high-value care, propose solutions to the identified situation, and pinpoint barriers to the implementation of effective solutions using a structured reflection framework and then participate in a debrief debate with fellow students. To assess the curriculum, reflective narratives were coded by frequency and codes were compared with one another and with relevant high-value care literature to identify patterns and themes. RESULTS: In total, 151 medical students participated in phase 1 and 119 in phase 3. For phase 2, 126 reflective narratives (94.7% participation rate) comprised 226 problems, 280 solutions, and 179 barriers. CONCLUSIONS: When provided appropriate resources, medical students are able to identify relevant examples of low-value care, downstream solutions, and barriers to implementation through a structured reflection curriculum comprising written narratives and in-person debate.